Methods and systems for monitoring and influencing gesture-based behaviors

ABSTRACT

Methods and systems are provided herein for analyzing, monitoring, and/or influencing a user&#39;s behavioral gesture in real-time. A gesture recognition method may be provided. The method may comprise: obtaining sensor data collected using at least one sensor located on a wearable device, wherein said wearable device is configured to be worn by a user; and analyzing the sensor data to determine a probability of the user performing a predefined gesture, wherein the probability is determined based in part on a magnitude of a motion vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 16/184,851, filed on Nov. 8, 2018, which is acontinuation of U.S. patent application Ser. No. 15/603,246, filed onMay 23, 2017, now U.S. Pat. No. 10,126,829, which is a continuation ofInternational Application Serial No. PCT/US2015/065862, filed on Dec.15, 2015, which application claims priority to U.S. ProvisionalApplication Ser. No. 62/092,283 filed on Dec. 16, 2014, all of which areincorporated herein by reference in their entirety.

BACKGROUND

Recent years have seen the proliferation of wearable devices such assmartwatches and wristbands in the consumer electronics market. Wearabledevices make computing technology pervasive by interweaving it intousers' daily lives. These wearable devices generally allow users totrack their fitness, activities, health and/or well-being, through theuse of electronics, software, and sensors in those devices.

Existing wearable devices are typically geared towards improvement ofusers' fitness and well-being. Additionally, the use of wearable devicescan be extended to other areas such as healthcare monitoring. Althoughwearable devices are capable of collecting large volumes of data aboutusers, there is presently a lack of systems and algorithms that canaccurately and efficiently analyze large volumes of data in certainhealthcare areas. Examples of those healthcare areas may includemonitoring of smoking behavior (e.g., smoking cessation), monitoring ofcertain types of eating and/or drinking disorders, monitoring of certaintypes of obsessive compulsive disorders, or monitoring of certain typesof neurological diseases that display symptoms associated withrepetitive vibration or shaking of a person's hands. Each of the abovebehaviors may be characterized by different and frequent ‘hand-to-mouth’gestures. Existing systems and algorithms often lack the capability toaccurately detect and monitor those gestures in real-time.

Thus, there is a need for methods and systems that can accurately detectand monitor various user gestures in real-time, and deliver relevant andpersonalized information to users in a timely manner to help them managecertain behaviors and habits, thus helping them improve their lives in astep-wise fashion.

SUMMARY

In some conventional systems, a plurality of physical motion profilepatterns may be stored in a library, and gesture recognition may becarried out by comparing the user's physical gesture (e.g., a shape ofthe gesture) against the plurality of motion profile patterns. However,this form of gesture recognition has several shortcomings. For example,body movement is different for different people and depends on a largenumber of parameters (e.g., body structure and its physical ratios,height, weight, posture (standing/sitting/driving), habits etc.). Thebody movement for each person may also vary at different times dependingon his/her mood and stress level, injuries, location (work/home/at a barwith friends), which hand is being used, time of day, etc. For example,in the case of smoking, people may smoke in different manners, differentbrands of cigarettes may be smoked differently, and the users' smokinghabits may change depending on which hand is being used, their moods,time of day, location etc. A cigarette as used herein may refer to anytype of tobacco products including, but not limited to, rolledcigarettes, cigarettes, e-cigarettes, cigars, and/or smoking pipes.

A significantly large number of permutations exist for different typesof body movement. To record and store a library of physical motionprofile patterns, and compare actual real-time gesture motions of alarge number of users to each physical motion profile pattern in thelibrary, would require an immense amount of memory storage and computingpower, which most mobile devices and wearable devices currently lack.Additionally, implementation of the above using cloud-based servers maynot be feasible in real-time given the high bandwidth required for datacommunication and processing.

Furthermore, physical motion profile patterns are typicallyfixed/generalized and do not account for subtle nuances in a user'sgesture. As a result, existing gesture recognition systems may not beable to detect whether a user is drinking a hot beverage or a coldbeverage, or smoking with a left hand or a right hand. Existing gesturerecognition systems also lack adaptability, and are generally unable tocapture and reflect changes in the user's gestures and/or behavior overtime.

In many instances, people may wish to improve their health andwell-being by reducing or eliminating certain types of undesirablebehaviors, for example smoking. Smoking is considered a significant riskfactor related to cancer and other diseases caused by inhalation oftobacco. Some smokers may embark on smoking cessation programs that areaimed at curing the addiction. However, studies have shown that althoughabout 50% of smokers have tried to quit smoking at some point in time,only 7% of them have successfully managed to do so. Most smokers aresusceptible to lapses during the course of those programs, eithersubconsciously or due to stress, peer pressure, or lack of self-control.In particular, smokers lack tools that can help them to monitor theirsmoking behavior, and that can proactively provide guidance in real-timeduring smoking lapses, to encourage them to put out a cigarette and stayon-course with a smoking cessation program.

Additionally, smoking involves unique and complex hand-to-mouth gesturesthat vary between smokers, depending on the type, size, and/or brand ofcigarette, a person's smoking history, gender, day and time of day ofsmoking, and a plethora of other factors. All these factors make itdifficult to track and filter out smoking gestures and patterns.

Accordingly, a need exists for systems and algorithms that can helpsmokers to control their smoking behaviors, reduce the number of smokedcigarettes, and set goals that are geared towards helping smokers toreduce or quit smoking. In particular, there is a need for systems andalgorithms that can accurately recognize hand-to-mouth gestures of auser and detect smoking lapses in real-time. A further need exists tomonitor a user's smoking behavior and predict when/where a user islikely to smoke, so that information (e.g., recommendations) can beprovided to help the user stay on course with a smoking cessationprogram and keep track of program goals. Such information may bepersonalized and dynamically provided in real-time to the user on acomputing device. The information can help the user to make informeddecisions about his/her overall well-being, and show the user theprogress that has been made. The systems and methods disclosed hereinaddress at least the above needs.

EMBODIMENT #1

A gesture recognition method may comprise: obtaining sensor datacollected using at least one sensor located on a wearable device,wherein said wearable device is configured to be worn by a user; andanalyzing the sensor data to determine a probability of the userperforming a predefined gesture, wherein the probability is determinedbased in part on a magnitude of a motion vector in the sensor data, andwithout comparing the motion vector to one or more physical motionprofiles.

EMBODIMENT #2

A system for implementing gesture recognition may comprise a memory forstoring sensor data collected using at least one sensor located on awearable device, wherein the wearable device is configured to be worn bya user. The system may further comprise one or more processorsconfigured to execute the set of software instructions to: analyze thesensor data to determine a probability of the user performing apredefined gesture, wherein the probability is determined based in parton a magnitude of a motion vector in the sensor data, and withoutcomparing the motion vector to one or more physical motion profiles.

EMBODIMENT #3

A tangible computer readable medium storing instructions that, whenexecuted by one or more processors, causes the one or more processors toperform a computer-implemented gesture recognition method, may beprovided. The method may comprise: obtaining sensor data collected usingat least one sensor located on a wearable device, wherein said wearabledevice is configured to be worn by a user; and analyzing the sensor datato determine a probability of the user performing a predefined gesture,wherein the probability is determined based in part on a magnitude of amotion vector in the sensor data, and without comparing the motionvector to one or more physical motion profiles.

In one or more of above embodiments #1, #2, and/or #3, the predefinedgesture may be selected from a group of different gestures associatedwith different activities. The gestures associated with the differentactivities may be differentiated from one another based at least on themagnitude of different motion vectors in the sensor data, and withoutcomparing the motion vectors to the one or more physical motionprofiles. The at least one sensor may comprise an accelerometer and agyroscope.

The magnitude of the motion vector may comprise: (1) a magnitude of theacceleration vector obtained from the accelerometer, and/or (2) amagnitude of an angular velocity vector obtained from the gyroscope. Theprobability may be determined based in part on the magnitude of theacceleration vector and/or the magnitude of the angular velocity vector.The probability may be determined based in part on the magnitude of theacceleration vector and/or the magnitude of the angular velocity vectorwithin different temporal periods, and without comparing theacceleration vector and/or the angular velocity vector to the one ormore physical motion profiles.

A pitch angle, a roll angle, and/or a yaw angle of the wearable devicemay be calculated based on the acceleration vector and/or the angularvelocity vector. The probability may be determined based on the pitchangle, the roll angle, and/or the yaw angle.

A correlation may be determined between the magnitude of theacceleration vector and the magnitude of the angular velocity vectorwithin different temporal periods, so as to determine the probability ofthe user performing the predefined gesture.

At least one sensor may further comprise one or more of the following: amagnetometer, a heart rate monitor, a global positioning system (GPS)receiver, an external temperature sensor, a microphone, a skintemperature sensor, a capacitive sensor, and/or a sensor configured todetect a galvanic skin response.

The sensor data may be analyzed without comparing the sensor dataagainst the one or more physical motion profiles. A shape of the one ormore physical motion profiles may be substantially similar to a shape ofone or more physical gestures of the user.

Analyzing the sensor data may further comprise calculating amulti-dimensional distribution function, wherein said multi-dimensionaldistribution function is a probability function of a plurality offeatures. The plurality of features may be associated with variousaspects of the predefined gesture. The plurality of features maycomprise two or more of the following features: (1) a time duration of asubmotion during the gesture; (2) the magnitude of the accelerationvector; (3) the magnitude of the angular velocity vector; (4) the rollangle; and (5) the pitch angle. The multi-dimensional distributionfunction may be associated with one or more characteristic of thepredefined gesture. The plurality of features may be encoded within thesensor data, and extracted from the sensor data. Two or more featuresmay be correlated.

The multi-dimensional distribution function may be configured to returna single probability value between 0 and 1, and wherein the probabilityvalue represents a probability of each feature. In some cases, eachfeature may be represented by a discrete value. In other cases, eachfeature may be measurable along a continuum. The multi-dimensionaldistribution function may be calculated by using Singular ValueDecomposition (SVD) to de-correlate the two or more correlated featuressuch that they are approximately orthogonal to each other. The use ofthe SVD may reduce a processing time required to compute the probabilityvalue for the multi-dimensional distribution function and may reduce theamount of sensor data needed to determine the probability of the userperforming the predefined gesture. The multi-dimensional distributionfunction may be calculated by multiplying the de-correlated (rotated) 1Dprobably density distribution of each feature, such that themulti-dimensional distribution function f(p₁, p₂, . . .p_(n))=f(p₁)*f(p₂)* . . . *f(p_(n)). The function f(p₁) may be a 1Dprobability density distribution of a first feature, the function f(p₂)may be a 1D probability density distribution of a second feature, andthe function f(p_(n)) may be a 1D probability density distribution of an-th feature. The 1D probability density distribution of each featuremay be obtained from a sample size of each feature. In some cases, thesample size may be constant across all of the features. In other cases,the sample size may be variable between different features. One or moreof the plurality of features may be determined whether they arestatistically insignificant. The one or more statistically insignificantfeatures may have low correlation with the predefined gesture. The oneor more statistically insignificant features may be removed from themulti-dimensional distribution function. Removing the one or morestatistically insignificant features from the multi-dimensionaldistribution function may reduce a computing time and/or power requiredto calculate the probability value for the multi-dimensionaldistribution function.

Analyzing the sensor data may further comprise applying a filter to thesensor data. The filter may be a higher order complex filter comprisinga finite-impulse-response (FIR) filter and/or aninfinite-impulse-response (IIR) filter. The filter may be a Kalmanfilter or a Parks-McClellan filter.

The wearable device may be configured to transmit the sensor data to auser device and/or a server for the analysis of the sensor data. Thetransmission of the sensor data may be via one or more wireless or wiredcommunication channels. The one or more wireless communication channelsmay comprise BLE (Bluetooth Low Energy), WiFi, 3G, and/or 4G networks.

The sensor data may be stored in a memory on the wearable device whenthe wearable device is not in operable communication with the userdevice and/or the server. The sensor data may be transmitted from thewearable device to the user device and/or the server when operablecommunication between the wearable device and the user device and/or theserver is re-established.

A data compression step may be applied to the sensor data. Thecompression of the sensor data may reduce a bandwidth required totransmit the sensor data, and the compression of the sensor data mayreduce a power consumption of the wearable device during thetransmission of the sensor data. The data compression step may comprisecalculating a time-based difference between samples of the sensor dataalong different axes of measurement. The time-based difference may betransmitted from the wearable device to a user device and/or a server.The sensor data may be compressed using a predefined number of bits.

The one or more sensors may be configured to collect the sensor data ata predetermined frequency. The predetermined frequency may be configuredto optimize and/or reduce a power consumption of the wearable device.The predetermined frequency may range from about 10 Hz to about 20 Hz.The one or more sensors may be configured to collect the sensor data ata first predetermined frequency when the probability that the user isperforming the gesture is below a predefined threshold value. The one ormore sensors may be configured to collect the sensor data at a secondpredetermined frequency when the probability that the user is performingthe gesture is above a predefined threshold value. The secondpredetermined frequency may be higher than the first predeterminedfrequency. The one or more sensors may be configured to collect thesensor data for a predetermined time duration. The one or more sensorsmay be configured to collect the sensor data continuously in real-timewhen the wearable device is powered on.

The one or more sensors may comprise a first group of sensors and asecond group of sensors. The first group of sensors and the second groupof sensors may be selectively activated to reduce power consumption ofthe wearable device. The first group of sensors and the second group ofsensors may be selectively activated to reduce an amount of thecollected sensor data. The reduction in the amount of sensor data mayallow for faster analysis/processing of the sensor data, and reduce anamount of memory required to store the sensor data. The first group ofsensors may be activated when the wearable device is powered on. Thefirst group of sensors may be used to determine the probability of theuser performing the predefined gesture. The second group of sensors maybe inactive when the probability that the user is performing the gestureis below a predefined threshold value. The second group of sensors maybe selectively activated when the wearable device is powered on and whenthe probability that the user is performing the gesture is above apredefined threshold value. The second group of sensors may beselectively activated upon determining that the user is performing thepredefined gesture. The second group of sensors may be activated tocollect additional sensor data, so as to confirm that the user isperforming the predefined gesture, monitor the gesture, and collectadditional sensor data relating to the gesture.

The wearable device may be configured to operate in a plurality ofenergy and/or performance modes. The plurality of modes may comprise alow power mode in which at least an accelerometer in the wearable deviceis turned on. The wearable device may have low power consumption whenthe wearable device is in the low power mode. An accuracy of detectionof the predefined gesture may be reduced when the wearable device is inthe low power mode, since less information (less amount of sensor data)is available for analysis in the low power mode. The plurality of modesmay comprise an accuracy mode in which all of the sensors are turned on.The wearable device may have high power consumption when the wearabledevice is in the accuracy mode. An accuracy of detection of thepredefined gesture may be improved when the wearable device is in theaccuracy mode, since more information (greater amount of sensor data) isavailable for analysis in the accuracy mode. In some cases, the sensordata may not be analyzed or transmitted when the wearable device is inan idle mode or a charging mode.

The sensor data may comprise at least one of the following parameters:(1) an active hand which the user uses to make the gesture; (2) a pulsepattern of the user; (3) a location of the user; (4) identifiers of thewearable device and/or a user device; and (5) behavioral statistics ofthe user relating to the gesture. An identity of the user may beauthenticated based on the one or more of the parameters. Theprobability of the user performing the predefined gesture at differenttimes of the day and/or at different geographical locations may bedetermined. A frequency of the sensor data collection may be adjustedbased on the different times of the day and/or the differentgeographical locations. The frequency of the sensor data collection maybe increased at times of the day and/or at geographical locations wherethe probability of the user performing the predefined gesture is above apredetermined threshold value. The frequency of the sensor datacollection may be reduced at times of the day and/or at geographicallocations where the probability of the user performing the predefinedgesture is below a predetermined threshold value. One or more of thesensors may be selectively activated based on the probability of theuser performing the predefined gesture at different times of the dayand/or at different geographical locations.

EMBODIMENT #4

A method of detecting a smoking gesture may comprise: obtaining sensordata collected using one or more sensors, wherein said sensors comprisea multi-axis accelerometer that is located on a wearable deviceconfigured to be worn by a user; and analyzing the sensor data todetermine a probability of the user smoking, wherein the probability isdetermined based in part on a magnitude of an acceleration vector in thesensor data, and without comparing the motion vector to one or morephysical motion profiles.

EMBODIMENT #5

A system for implementing gesture recognition may comprise a memory forstoring sensor data collected using one or more sensors, wherein thesensors may comprise a multi-axis accelerometer that is located on awearable device configured to be worn by a user. The system may furthercomprise one or more processors configured to execute a set of softwareinstructions to: analyze the sensor data to determine a probability ofthe user smoking, wherein the probability is determined based in part ona magnitude of an acceleration vector in the sensor data, and withoutcomparing the motion vector to one or more physical motion profiles.

EMBODIMENT #6

A tangible computer readable medium storing instructions that, whenexecuted by one or more processors, causes the one or more processors toperform a computer-implemented gesture recognition method, may beprovided. The method may comprise: obtaining sensor data collected usingone or more sensors, wherein said sensors comprise a multi-axisaccelerometer that is located on a wearable device configured to be wornby a user; and analyzing the sensor data to determine a probability ofthe user smoking, wherein the probability is determined based in part ona magnitude of an acceleration vector in the sensor data, and withoutcomparing the motion vector to one or more physical motion profiles.

In one or more of above embodiments #4, #5, and/or #6, analyzing thesensor data may comprise analyzing one or more features in the sensordata to determine a probability of the user taking a cigarette puff. Thefeatures may comprise at least one of the following: (1) a time durationthat a potential cigarette is detected in the user's mouth; (2) a rollangle of the user's arm; (3) a pitch angle of the smoker's arm; (4) atime duration of a potential smoking puff; (5) a time duration betweenconsecutive potential puffs; (6) number of potential puffs that the usertakes to finish smoking a cigarette; (7) the magnitude of theacceleration vector; (8) a speed of the user's arm; (9) an inhale regioncorresponding to an arm-to-mouth gesture; and/or (10) an exhale regioncorresponding to an arm-down-from-mouth gesture. The features may beextracted from the sensor data.

The probability of the user smoking may be adjusted based on one or moreuser inputs. The user inputs may comprise: (1) an input signalindicating that the user did not smoke; (2) an input signal indicatingthat the user had smoked; and (3) an input signal indicating that theuser had smoked but the smoking gesture was not recognized or detected.A user configuration file (UCF) for the user may be generated based onthe analyzed sensor data and the one or more user inputs. The UCF may begeneral to a plurality of users. The UCF may become unique to each userafter a period of time. The UCF may be configured to adapt and changeover time depending on the user's behavior. The UCF may comprise a listof user parameters associated with different activities besides smoking.The different activities may comprise at least one of the following:standing, walking, sitting, driving, drinking, eating, and/or leaningwhile standing or sitting. The UCF may be dynamically changed when nosmoking of the user has been detected for a predetermined time period.The UCF may be dynamically changed to verify that the user has notsmoked for the predetermined time period.

The user may be determined whether to be smoking with a right hand or aleft hand based on a roll angle, a pitch angle, and/or a yaw angleextracted from the sensor data. The UCF may be updated with theleft/right hand information of the user.

The probability may be determined using a multi-dimensional distributionfunction that is associated with one or more smoking characteristics.The one or more smoking characteristics may comprise the user taking oneor more cigarette puffs. The multi-dimensional distribution function maybe generated for each puff. The probability of the user smoking may bedetermined based on: (1) a number of potential puffs; (2) themulti-dimensional distribution function for each potential puff; and (3)a time duration in which the number of potential puffs occur. A sum ofthe multi-dimensional distribution functions for a number of potentialpuffs may be determined whether to be equal to or greater than apredetermined probability threshold. The user may be determined to besmoking when the sum is equal to or greater than the predeterminedprobability threshold, and the user may be determined not to be smokingwhen the sum is less than the predetermined probability threshold. Theuser may be determined to be smoking when a predetermined number ofpuffs have been detected within a predetermined time period. The rolland pitch angles associated with the potential puffs may be analyzed,and the puffs whose roll and pitch angles fall outside of apredetermined roll/pitch threshold may be discarded. A time durationbetween the potential puffs may be analyzed, and the puffs where thetime duration falls outside of a predetermined time period may bediscarded.

The probability of the user smoking at different times of the day and/orat different geographical locations may be determined. A frequency ofthe sensor data collection may be adjusted based on the different timesof the day and/or the different geographical locations. The frequency ofthe sensor data collection may be increased at times of the day and/orat geographical locations where the probability of the user smoking isabove a predetermined threshold value. The frequency of the sensor datacollection may be reduced at times of the day and/or at geographicallocations where the probability of the user smoking is below apredetermined threshold value. One or more of the sensors may beselectively activated based on the probability of the user smoking atdifferent times of the day and/or at different geographical locations.

It shall be understood that different aspects of the disclosure can beappreciated individually, collectively, or in combination with eachother. Various aspects of the disclosure described herein may be appliedto any of the particular applications set forth below or for any othertypes of energy monitoring systems and methods.

Other objects and features of the present disclosure will becomeapparent by a review of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the disclosure are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 illustrates a healthcare monitoring system in accordance withsome embodiments;

FIG. 2 illustrates exemplary components in a healthcare monitoringsystem, in accordance with some embodiments;

FIG. 3 illustrates the determination of a pitch angle, a roll angle,and/or a yaw angle of a wearable device based on sensor data from agyroscope and/or an accelerometer on the wearable device, in accordancewith some embodiments;

FIG. 4 illustrates the correlation in magnitude of an angular velocityvector to a reference as a user is smoking, in accordance with someembodiments;

FIG. 5 illustrates the correlation in magnitudes of the accelerationvector and the angular velocity vector as a user is eating, inaccordance with some embodiments;

FIG. 6 illustrates the correlation in magnitudes of the accelerationvector and the angular velocity vector as a user is brushing teeth, inaccordance with some embodiments;

FIG. 7 illustrates the correlation in magnitudes of the accelerationvector and the angular velocity vector as a user is drinking a colddrink, in accordance with some embodiments;

FIG. 8 illustrates the correlation in magnitudes of the accelerationvector and the angular velocity vector during a smoking episode (for asingle puff), in accordance with some embodiments;

FIG. 9 is graph of the probability that a user is smoking a cigarette asa function of number of smoking puffs, in accordance with someembodiments;

FIG. 10 is a flowchart of a method of detecting a probability of a usersmoking a cigarette, in accordance with some embodiments;

FIG. 11 is a flowchart of a method of detecting a probability of a usersmoking a cigarette, in accordance with some other embodiments;

FIG. 12 is a flowchart of a method of detecting a probability of a usersmoking a cigarette, in accordance with some further embodiments;

FIG. 13 illustrates an exemplary window depicting the number ofcigarettes smoked during a day by a user, in accordance with someembodiments;

FIG. 14 illustrates an exemplary window depicting the breakdown ofcigarettes smoked by time of day, in accordance with some embodiments;

FIG. 15 illustrates an exemplary window depicting the number ofcigarettes smoked by a user on weekdays and weekends during a week, inaccordance with some embodiments;

FIG. 16 illustrates an exemplary window depicting the breakdown ofcigarettes smoked by time of day over a period of four weeks, inaccordance with some embodiments;

FIG. 17 illustrates an exemplary window depicting a user's daily goal,in accordance with some embodiments;

FIG. 18 illustrates an exemplary window ranking a smoker's cessationsuccess/performance against other smokers in a group, in accordance withsome embodiments; and

FIG. 19 illustrates an exemplary window showing a plurality of smokingmetrics of a user, in accordance with some embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to some exemplary embodiments ofthe disclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings and disclosure to refer to the same or likeparts.

Introduction

Wearable devices have become increasingly popular in recent years.Although wearable devices are capable of collecting large volumes ofdata about users, there is presently a lack of systems and algorithmsthat can accurately and efficiently analyze large volumes of data,particularly in certain healthcare areas.

The embodiments of the disclosure described herein can enable real-timemeasurement of hand-to-mouth gestures applicable to certain healthcareareas (e.g., monitoring of smoking behavior for smoking cessation,etc.). The data can be used to help users effectively manage or controltheir behavior/habits. In some cases, the data may be used by healthcareorganizations or insurance companies to tailor preventive behavioralhealth programs for users, that can help users to improve their healthand well-being.

Embodiments of the disclosure can help users to better understand theirbehaviors/habits in order to more effectively change theirbehaviors/habits. For example, in the case of smoking, certainembodiments of the disclosure allow users to see the number ofcigarettes smoked by time and location, number of puffs, the socialcontext in which smoking occurred, etc. Smoking statistics and alertscan be generated for different users, and goals may be set for cessationprograms. A user can also compare his progress in a smoking cessationprogram to other users in a same social network. In some cases,incentives may be provided to users in real-time to reward progress andto further motivate the user.

Next, various embodiments of the disclosure will be described withreference to the drawings.

FIG. 1 illustrates a healthcare monitoring system in accordance withsome embodiments. In one aspect, a healthcare monitoring system 100 mayinclude a user device 102, a wearable device 104, a server 106, agesture analysis engine 108, and a database 110. Each of the components102, 104, 106, 108, and 110 may be operatively connected to one anothervia network 112 or any type of communication links that allowstransmission of data from one component to another. The gesture analysisengine may be configured to analyze input data from the user deviceand/or wearable device in order to detect and/or monitor a predeterminedgesture, and to provide information (e.g., recommendations) to assist auser in managing behavior associated with the predetermined gesture. Thegesture analysis engine may be implemented anywhere within thehealthcare monitoring system, and/or outside of the healthcaremonitoring system. In some embodiments, the gesture analysis engine maybe implemented on the server. In other embodiments, the gesture analysisengine may be implemented on the user device. Additionally, the gestureanalysis engine may be implemented on the wearable device. In somefurther embodiments, a plurality of gesture analysis engines may beimplemented on one or more servers, user devices, and/or wearabledevices. Alternatively, the gesture analysis engine may be implementedin one or more databases. The gesture analysis engine may be implementedusing software, hardware, or a combination of software and hardware inone or more of the above-mentioned components within the healthcaremonitoring system.

User device 102 may be a computing device configured to perform one ormore operations consistent with the disclosed embodiments. Examples ofuser devices may include, but are not limited to, mobile devices,smartphones/cellphones, tablets, personal digital assistants (PDAs),laptop or notebook computers, desktop computers, media content players,television sets, video gaming station/system, virtual reality systems,augmented reality systems, microphones, or any electronic device capableof analyzing, receiving, providing or displaying certain types ofbehavioral data (e.g., smoking data) to a user. The user device may be ahandheld object. The user device may be portable. The user device may becarried by a human user. In some cases, the user device may be locatedremotely from a human user, and the user can control the user deviceusing wireless and/or wired communications.

User device 102 may include one or more processors that are capable ofexecuting non-transitory computer readable media that may provideinstructions for one or more operations consistent with the disclosedembodiments. The user device may include one or more memory storagedevices comprising non-transitory computer readable media includingcode, logic, or instructions for performing the one or more operations.The user device may include software applications that allow the userdevice to communicate with and transfer data between wearable device104, server 106, gesture analysis engine 108, and/or database 110. Theuser device may include a communication unit, which may permit thecommunications with one or more other components in healthcaremonitoring system 100. In some instances, the communication unit mayinclude a single communication module, or multiple communicationmodules. In some instances, the user device may be capable ofinteracting with one or more components in healthcare monitoring system100 using a single communication link or multiple different types ofcommunication links.

User device 102 may include a display. The display may be a screen. Thedisplay may or may not be a touchscreen. The display may be alight-emitting diode (LED) screen, OLED screen, liquid crystal display(LCD) screen, plasma screen, or any other type of screen. The displaymay be configured to show a user interface (UI) or a graphical userinterface (GUI) rendered through an application (e.g., via anapplication programming interface (API) executed on the user device).The GUI may show images that permit a user to monitor one or more typesof behavior (e.g., smoking). The user device may also be configured todisplay webpages and/or websites on the Internet. One or more of thewebpages/websites may be hosted by server 106 and/or rendered by gestureanalysis engine 108.

A user may navigate within the GUI through the application. For example,the user may select a link by directly touching the screen (e.g.,touchscreen). The user may touch any portion of the screen by touching apoint on the screen. Alternatively, the user may select a portion of animage with aid of a user interactive device (e.g., mouse, joystick,keyboard, trackball, touchpad, button, verbal commands,gesture-recognition, attitude sensor, thermal sensor, touch-capacitivesensors, or any other device). A touchscreen may be configured to detectlocation of the user's touch, length of touch, pressure of touch, and/ortouch motion, whereby each of the aforementioned manner of touch may beindicative of a specific input command from the user.

Wearable device 104 may include smartwatches, wristbands, glasses,gloves, headgear (such as hats, helmets, virtual reality headsets,augmented reality headsets, head-mounted devices (HMD), headbands),pendants, armbands, leg bands, shoes, vests, motion sensing devices,etc. The wearable device may be configured to be worn on a part of auser's body (e.g., a smartwatch or wristband may be worn on the user'swrist). The wearable device may include one or more types of sensors.Examples of types of sensors may include inertial sensors (e.g.,accelerometers, gyroscopes, and/or gravity detection sensors, which mayform inertial measurement units (IMUs)), location sensors (e.g., globalpositioning system (GPS) sensors, mobile device transmitters enablinglocation triangulation), heart rate monitors, external temperaturesensors, skin temperature sensors, capacitive touch sensors, sensorsconfigured to detect a galvanic skin response (GSR), vision sensors(e.g., imaging devices capable of detecting visible, infrared, orultraviolet light, such as cameras), proximity or range sensors (e.g.,ultrasonic sensors, lidar, time-of-flight or depth cameras), altitudesensors, attitude sensors (e.g., compasses), pressure sensors (e.g.,barometers), humidity sensors, vibration sensors, audio sensors (e.g.,microphones), and/or field sensors (e.g., magnetometers, electromagneticsensors, radio sensors).

Wearable device 104 may further include one or more devices capable ofemitting a signal into an environment. For instance, the wearable devicemay include an emitter along an electromagnetic spectrum (e.g., visiblelight emitter, ultraviolet emitter, infrared emitter). The wearabledevice may include a laser or any other type of electromagnetic emitter.The wearable device may emit one or more vibrations, such as ultrasonicsignals. The wearable device may emit audible sounds (e.g., from aspeaker). The wearable device may emit wireless signals, such as radiosignals or other types of signals.

Any examples herein of sensors that may be present in wearable device104 may also apply to user device 102. For instance, one or moredifferent sensors may be incorporated into user device 102.

Although FIG. 1 illustrates user device 102 and wearable device 104 astwo separate devices, the disclosure is not limited thereto. In someembodiments, the user device and the wearable device may be integratedinto a single device. In some embodiments, the wearable device may beincorporated into the user device. In other embodiments, the user devicemay be incorporated into the wearable device. Alternatively, the userdevice may be capable of performing one or more functions of thewearable device. Optionally, the wearable device may be capable ofperforming one or more functions of the user device, and the user devicemay be capable of performing one or more functions of the wearabledevice.

User device 102 and wearable device 104 may be operated by one or moreusers consistent with the disclosed embodiments. In some embodiments, auser may be associated with a unique user device and a unique wearabledevice. Alternatively, a user may be associated with a plurality of userdevices and wearable devices. A user as described herein may refer to anindividual or a group of individuals who are seeking to improve theirwell-being through healthcare monitoring system 100. For example, asmoker or a group of smokers may desire to quit smoking. A person or agroup of persons suffering from alcoholism may desire to quit drinking.A person or a group of persons suffering from an excessive eatingdisorder may desire to reduce their food intake. The above users can usehealthcare monitoring system 100 to control and manage those behaviors.

A user may be registered or associated with an entity that providesservices associated with one or more operations performed by thedisclosed embodiments. For example, a user may be a registered user ofan entity (e.g., a company, an organization, an individual, etc.) thatprovides gesture analysis engine 108 to perform operations for assistingthe user in managing certain types of behaviors (e.g., smoking). Thedisclosed embodiments are not limited to any specific relationships oraffiliations between user(s) of user device 102 and wearable device 104,and an entity, person(s), or entities that provides gesture analysisengine 108.

User device 102 and/or wearable device 104 may be configured to receiveinput from one or more users. A user may provide an input to the userdevice and/or wearable device using an input device, for example, akeyboard, a mouse, a touch-screen panel, voice recognition and/ordictation software, or any combination of the above. The user input mayinclude statements, comments, questions, or answers relating to certaintypes of behavior (e.g., smoking). Different users may provide differentinputs. For example, a user may provide an input to indicate whether theuse is smoking or had smoked within a predetermined time period. In someinstances, the input may also indicate how the user is feeling (e.g.,whether the user is feeling motivated or discouraged) during the courseof a program aimed at mitigating certain behaviors (e.g., smoking). Inthose instances, the user's input may be indicative of the user'sthoughts, feelings, moods, opinions, questions, and/or answers relatingto smoking.

Server 106 may be one or more server computers configured to perform oneor more operations consistent with the disclosed embodiments. In oneaspect, the server may be implemented as a single computer, throughwhich user device 102 and wearable device 104 are able to communicatewith gesture analysis engine 108 and database 110. In some embodiments,the user device and/or the wearable device may communicate with thegesture analysis engine directly through the network. In someembodiments, the server may communicate on behalf of the user deviceand/or the wearable device with the gesture analysis engine or databasethrough the network. In some embodiments, the server may embody thefunctionality of one or more of gesture analysis engines. In someembodiments, one or more gesture analysis engines may be implementedinside and/or outside of the server. For example, the gesture analysisengines may be software and/or hardware components included with theserver or remote from the server.

In some embodiments, the user device and/or the wearable device may bedirectly connected to the server through a separate link (not shown inFIG. 1). In certain embodiments, the server may be configured to operateas a front-end device configured to provide access to one or moregesture analysis engines consistent with certain disclosed embodiments.The server may, in some embodiments, utilize one or more gestureanalysis engines to analyze input data from the user device and/orwearable device in order to detect and/or monitor a predeterminedgesture, and to provide information (e.g., recommendations) to assistthe user in managing behavior associated with the predetermined gesture.The server may also be configured to store, search, retrieve, and/oranalyze data and information stored in one or more of the databases. Thedata and information may include raw data collected from accelerometersand gyroscopes on one or more wearable devices, as well as each user'shistorical behavioral pattern and social interactions relating to thetype of behavior (e.g., smoking). While FIG. 1 illustrates the server asa single server, in some embodiments, multiple devices may implement thefunctionality associated with a server.

A server may include a web server, an enterprise server, or any othertype of computer server, and can be computer programmed to acceptrequests (e.g., HTTP, or other protocols that can initiate datatransmission) from a computing device (e.g., user device and/or wearabledevice) and to serve the computing device with requested data. Inaddition, a server can be a broadcasting facility, such as free-to-air,cable, satellite, and other broadcasting facility, for distributingdata. A server may also be a server in a data network (e.g., a cloudcomputing network).

A server may include known computing components, such as one or moreprocessors, one or more memory devices storing software instructionsexecuted by the processor(s), and data. A server can have one or moreprocessors and at least one memory for storing program instructions. Theprocessor(s) can be a single or multiple microprocessors, fieldprogrammable gate arrays (FPGAs), or digital signal processors (DSPs)capable of executing particular sets of instructions. Computer-readableinstructions can be stored on a tangible non-transitorycomputer-readable medium, such as a flexible disk, a hard disk, a CD-ROM(compact disk-read only memory), and MO (magneto-optical), a DVD-ROM(digital versatile disk-read only memory), a DVD RAM (digital versatiledisk-random access memory), or a semiconductor memory. Alternatively,the methods can be implemented in hardware components or combinations ofhardware and software such as, for example, ASICs, special purposecomputers, or general purpose computers.

While FIG. 1 illustrates the server as a single server, in someembodiments, multiple devices may implement the functionality associatedwith server.

Network 112 may be a network that is configured to provide communicationbetween the various components illustrated in FIG. 1. The network may beimplemented, in some embodiments, as one or more networks that connectdevices and/or components in the network layout for allowingcommunication between them. For example, user device 102, wearabledevice 104, and gesture analysis engine 108 may be in operablecommunication with one another over network 112. Direct communicationsmay be provided between two or more of the above components. The directcommunications may occur without requiring any intermediary device ornetwork. Indirect communications may be provided between two or more ofthe above components. The indirect communications may occur with aid ofone or more intermediary device or network. For instance, indirectcommunications may utilize a telecommunications network. Indirectcommunications may be performed with aid of one or more router,communication tower, satellite, or any other intermediary device ornetwork. Examples of types of communications may include, but are notlimited to: communications via the Internet, Local Area Networks (LANs),Wide Area Networks (WANs), Bluetooth, Near Field Communication (NFC)technologies, networks based on mobile data protocols such as GeneralPacket Radio Services (GPRS), GSM, Enhanced Data GSM Environment (EDGE),3G, 4G, or Long Term Evolution (LTE) protocols, Infra-Red (IR)communication technologies, and/or Wi-Fi, and may be wireless, wired, ora combination thereof. In some embodiments, the network may beimplemented using cell and/or pager networks, satellite, licensed radio,or a combination of licensed and unlicensed radio. The network may bewireless, wired, or a combination thereof

User device 102, wearable device 104, server 106, and/or gestureanalysis engine 110 may be connected or interconnected to one or moredatabases 110. The databases may be one or more memory devicesconfigured to store data. Additionally, the databases may also, in someembodiments, be implemented as a computer system with a storage device.In one aspect, the databases may be used by components of the networklayout to perform one or more operations consistent with the disclosedembodiments.

In one embodiment, the databases may comprise storage containing avariety of data sets consistent with disclosed embodiments. For example,the databases may include, for example, raw data collected byaccelerometers and gyroscopes located on wearable device 104. Thedatabases may also include users' preferences, historical behavioralpatterns, traits associated with a type of behavior, changes and/orimprovements in the users' lifestyles, the users' success at managing orovercoming certain types of behaviors, the users' social interactionrelating to a certain type of behavior, statements or commentsindicative of how the user is feeling at different points in time, etc.In some embodiments, the database(s) may include crowd-sourced datacomprising comments and insights relating to user's attempts to quitsmoking obtained from internet forums and social media websites. TheInternet forums and social media websites may include personal and/orgroup blogs, Facebook™, Twitter™, etc. Additionally, in someembodiments, the database(s) may include crowd-sourced data comprisingcomments and insights relating to other users' attempts to quit smoking,whereby those comments and insights may be directly input by one or moreother users into the gesture analysis engine(s). The crowd-sourced datamay contain up-to-date or current information on the progress of otherusers in trying to quit smoking, recommendations on ways to quitsmoking, etc. The crowd-sourced data may be provided by other users whohave experience with trying to quit smoking, or who have successfullymanaged to quit smoking.

In certain embodiments, one or more of the databases may be co-locatedwith the server, may be co-located with one another on the network, ormay be located separately from other devices (signified by the dashedline connecting the database(s) to the network). One of ordinary skillwill recognize that the disclosed embodiments are not limited to theconfiguration and/or arrangement of the database(s).

Any of the user device, wearable device, server, gesture analysisengine, and the database may, in some embodiments, be implemented as acomputer system. Additionally, while the network is shown in FIG. 1 as a“central” point for communications between components, the disclosedembodiments are not so limited. For example, one or more components ofthe network layout may be interconnected in a variety of ways, and mayin some embodiments be directly connected to, co-located with, or remotefrom one another, as one of ordinary skill will appreciate.Additionally, while some disclosed embodiments may be implemented on theserver, the disclosed embodiments are not so limited. For instance, insome embodiments, other devices (such as gesture analysis system(s)and/or database(s)) may be configured to perform one or more of theprocesses and functionalities consistent with the disclosed embodiments,including embodiments described with respect to the server.

Although particular computing devices are illustrated and networksdescribed, it is to be appreciated and understood that other computingdevices and networks can be utilized without departing from the spiritand scope of the embodiments described herein. In addition, one or morecomponents of the network layout may be interconnected in a variety ofways, and may in some embodiments be directly connected to, co-locatedwith, or remote from one another, as one of ordinary skill willappreciate.

The gesture analysis engine(s) may be implemented as one or morecomputers storing instructions that, when executed by processor(s),analyze input data from a user device and/or a wearable device in orderto detect and/or monitor a predetermined gesture, and to provideinformation (e.g., recommendations) to assist the user in managingbehavior associated with the predetermined gesture. The gesture analysisengine(s) may also be configured to store, search, retrieve, and/oranalyze data and information stored in one or more databases. The dataand information may include raw data collected from accelerometers andgyroscopes on one or more wearable devices, as well as each user'shistorical behavioral pattern and social interactions relating to thetype of behavior (e.g., smoking). In some embodiments, server 106 may bea computer in which the gesture analysis engine is implemented.

However, in some embodiments, one or more gesture analysis engine(s) 108may be implemented remotely from server 106. For example, a user devicemay send a user input to server 106, and the server may connect to oneor more gesture analysis engine(s) 108 over network 112 to retrieve,filter, and analyze data from one or more remotely located database(s)110. In other embodiments, the gesture analysis engine(s) may representsoftware that, when executed by one or more processors, performprocesses for analyzing data to detect and/or monitor a predeterminedgesture, and to provide information (e.g., recommendations) to assistthe user in managing or overcoming certain types of behaviors.

A server may access and execute gesture analysis engine(s) to performone or more processes consistent with the disclosed embodiments. Incertain configurations, the gesture analysis engine(s) may be softwarestored in memory accessible by a server (e.g., in memory local to theserver or remote memory accessible over a communication link, such asthe network). Thus, in certain aspects, the gesture analysis engine(s)may be implemented as one or more computers, as software stored on amemory device accessible by the server, or a combination thereof. Forexample, one gesture analysis engine(s) may be a computer executing oneor more gesture recognition techniques, and another gesture analysisengine(s) may be software that, when executed by a server, performs oneor more gesture recognition techniques.

The functions of the gesture analysis engine, and its communication withthe user device and wearable device, will be described in detail belowwith reference to FIG. 2. Although various embodiments are describedherein using monitoring or cessation of smoking behavior as an example,it should be noted that the disclosure is not limited thereto, and canbe used to monitor other types of behaviors and activities besidessmoking.

FIG. 2 illustrates exemplary components in a healthcare monitoringsystem in accordance with some embodiments. Referring to FIG. 2, ahealthcare monitoring system 200 may comprise a user device 102, awearable device 104, and a gesture analysis engine 108. As previouslydescribed, the gesture analysis engine may be implemented both insideand/or outside of a server. For example, the gesture analysis engine maybe software and/or hardware components included with a server, or remotefrom the server. In some embodiments, the gesture analysis engine (orone or more functions of the gesture analysis engine) may be implementedon the user device and/or wearable device. Alternatively, the userdevice, wearable device, and/or server may be configured to performdifferent functions of the gesture analysis engine. Optionally, one ormore functions of the gesture analysis engine may be duplicated acrossthe user device, wearable device, and/or server.

In the example of FIG. 2, wearable device 104 may comprise at least onesensor 105. For example, the wearable device may comprise anaccelerometer 105-1 and a gyroscope 105-2. One or more other types ofsensors as described elsewhere herein may be incorporated into thewearable device.

The user device and/or the wearable device may be configured to provideinput data 114 to the gesture analysis engine. The input data maycomprise sensor data 114 a, user input 114 b, user location 114 c,historical behavioral data 114 d, and social network interaction data114 e.

The sensor data may comprise raw data collected by the accelerometer andthe gyroscope on the wearable device. The sensor data may be stored inmemory located on the wearable device, user device, and/or server. Insome embodiments, the sensor data may be stored in one or moredatabases. The databases may be located on the server, user device,and/or wearable device. Alternatively, the databases may be locatedremotely from the server, user device, and/or wearable device.

The user input may be provided by a user via the user device and/or thewearable device. The user input may be in response to questions providedby the gesture analysis engine. Examples of questions may includewhether the user is currently smoking, whether the user had smokedwithin a predetermined time period (e.g., within the last 8 hours),number of cigarettes smoked within a predetermined time period, time andplace where the user had smoked, brand of cigarette, whether the userhad planned to smoke that cigarette, whether the user is smoking aloneor with others, how the user is feeling at a particular moment withrelation to smoking, etc. The user's responses to those questions may beused to supplement the sensor data to determine the probability of acurrent or future smoking episode, and predict where/when the user islikely to smoke. This information obtained from the user input can beanalyzed using machine learning processes.

The user location may be determined by a location sensor (e.g., GPSreceiver) on the user device and/or the wearable device. The userlocation may be used to determine places where the user is smoking or islikely to smoke. The user location may also be used to supplement thesensor data to determine the probability of a current or future smokingepisode.

The historical behavioral data may correspond to smoking-related datacollected over a predetermined time period. The historical behavioraldata may be stored in memory located on the wearable device, userdevice, and/or server. In some embodiments, the historical behavioraldata may be stored in one or more databases. The databases may belocated on the server, user device, and/or wearable device.Alternatively, the databases may be located remotely from the server,user device, and/or wearable device.

The social network interaction data may be obtained from an application(e.g., a mobile application) provided by the gesture analysis engine.The application may allow a user to pick a social group within theapplication and to compare his/her performance to other users in thesocial group. The social group may be defined by the users. The users inthe social group may be seeking to manage or control a certain type ofbehavior or habit (e.g., smoking) using the application. The user'sperformance may include the user's successes and/or failures in managingthe type of behavior or habit, compared to other users in the group.

The gesture analysis engine may be configured to obtain sensor data fromat least one sensor located on the wearable device and/or the userdevice. For example, the gesture analysis engine may be configured toobtain sensor data from the accelerometer and/or gyroscope located onthe wearable device. As previously mentioned, the wearable device may beworn by a user (e.g., on the user's wrist). The gesture analysis enginemay be further configured to analyze the sensor data to determine aprobability of the user performing a predefined gesture. The probabilitymay be determined based in part on a magnitude of a motion vector in thesensor data, and without comparing the motion vector to one or morephysical motion profiles.

Types of Gestures

A predefined gesture may be associated with and unique to at least oneof the following activities: smoking, drinking, eating, shaving,brushing teeth, nails biting, vomiting, or chronic coughing. The gestureanalysis engine may be configured to determine a probability of a userperforming one or more of the above activities. The gestures associatedwith different activities may be differentiated from one another basedat least on the magnitude of different motion vectors in the sensordata, and without comparing the motion vectors to one or more physicalmotion profiles.

In some cases, the gesture analysis engine may be capable of determiningwhether a user is drinking a hot liquid or drinking a cold liquid, basedon the magnitudes of the motion vectors in the sensor data and theirtime durations. In other cases, the gesture analysis engine may becapable of determining a user's consumption of different types of foods,based on the magnitudes of the motion vectors in the sensor data andtheir time durations. For example, the gesture analysis engine canidentify a hot drink or a cold drink based on number of sips, sipduration, roll, pitch etc. The gesture analysis engine can also identifythe general type of food that is being consumed (e.g., whether the useris drinking soup with a spoon, eating solid food with knife and fork,eating snacks, using chopsticks, etc.). Accordingly, the gestureanalysis engine may be capable of detecting various subcategories withineach activity.

In some cases, the predefined gesture may be associated with aneurological disorder that causes repetitive movement and/or vibrationof an extremity of the user. The gesture analysis engine may beconfigured to determine a probability of a user performing thepredefined gesture, so as to determine an extent to which the user issuffering from the neurological disorder.

Sensor Data

As previously mentioned, an accelerometer may be disposed on thewearable device. In some embodiments, the accelerometer may be amulti-axis accelerometer such as an n-axis accelerometer, whereby n maybe an integer that is equal to or greater than 2. For example, in someembodiments, the accelerometer may be a 3-axis accelerometer. Theaccelerometer may be able of measuring acceleration along an X-axis, aY-axis, and a Z-axis in a local coordinate system that is definedrelative to the wearable device.

Accordingly, in those embodiments, the motion vector in the sensor datamay be an acceleration vector, and the magnitude of the motion vectormay correspond to a magnitude of the acceleration vector. Theacceleration vector may comprise a plurality of acceleration componentsmeasured along different axes of the accelerometer. For example, theplurality of acceleration components along the X-axis, Y-axis, andZ-axis may be given by Ax, Ay, and Az, respectively. Each of theacceleration components may be a vector quantity. The magnitude of theacceleration vector may be given by A_(m)=SQRT (A_(x) ²+A_(y) ²+A_(z)²). The magnitude of the acceleration vector may be a scalar quantity.

The gesture analysis engine may be configured to determine theprobability of the user performing the predefined gesture based in parton the magnitude of the accelerator vector. For example, the gestureanalysis engine may be configured to determine the probability based inpart on the magnitude of the acceleration vector within differenttemporal periods, and without comparing the acceleration vector (and/oreach acceleration component in the acceleration vector) to one or morephysical motion profiles.

In some embodiments, a pitch angle and/or a roll angle of the wearabledevice may be calculated by the gesture analysis engine using theacceleration components along the X-axis, Y-axis, and Z-axis. In somecases, the pitch angle may be given by θ=arctan [A_(y)/SQRT (A_(x)²+A_(z) ²)]. In some embodiments, a roll angle of the wearable devicemay be calculated using the acceleration components along the X-axis andZ-axis. In some cases, the roll angle may be given by ϕ=arctan[−A_(x)/A_(z)]. The pitch angle and the roll angle may be indicative ofa rotational motion of a portion of the user's body (where the wearabledevice is worn) about the Y-axis and the X-axis, respectively. In someembodiments, the gesture analysis engine may be configured to determinethe probability of the user performing the predefined gesture based inpart of the pitch angle and/or the roll angle.

As previously mentioned, a gyroscope may also be disposed on thewearable device. In those embodiments, the motion vector in the sensordata may be an angular velocity vector, and the magnitude of the motionvector may correspond to a magnitude of the angular velocity vector. Theangular velocity vector may comprise a plurality of angular velocitycomponents measured along different axes of the gyroscope.

The gesture analysis engine may be configured to determine theprobability of the user performing the predefined gesture based in parton the magnitude of the angular velocity vector, and without comparingthe angular velocity vector to one or more physical motion profiles. Forexample, the gesture analysis engine may be configured to determine theprobability based in part on the magnitude of the angular velocityvector within different temporal periods.

In some embodiments, as shown in FIG. 3, a pitch angle, a roll angle,and/or a yaw angle of the wearable device may be determined based onsensor data from the gyroscope and/or the accelerometer on the wearabledevice. The pitch angle, the roll angle, and/or the yaw angle may beindicative of a rotational motion of a part of the user's body about anX-axis, a Y-axis, and a Z-axis in a local coordinate system that isdefined on the wearable device. In FIG. 3, the pitch angle may beindicative of a rotational motion of the user's wrist about the Y-axis,the roll angle may be indicative of a rotational motion of the user'swrist about the X-axis, and the yaw angle may be indicative of arotational motion of the user's wrist about the Z-axis. The sensor datafrom the wearable device may be transmitted to the user device, whichmay subsequently transmit the sensor data to the gesture analysisengine. In some embodiments, the sensor data from the wearable devicemay be transmitted directly to the gesture analysis engine without goingthrough the user device. The gesture analysis engine may be configuredto determine the probability of the user performing the predefinedgesture based in part on the pitch angle, the roll angle, and/or the yawangle.

In some embodiments, the gesture analysis engine may be configured todetermine the probability of the user performing the predefined gesturebased in part on the magnitude of the acceleration vector and themagnitude of the angular velocity vector, and without comparing theacceleration vector and the angular velocity vector to one or morephysical motion profiles. As previously mentioned, the predefinedgesture may be associated with and unique to activities such as smoking,drinking, brushing teeth, and drinking. The gesture analysis engine maybe configured to determine a probability of a user performing one ormore of the above activities. The gestures associated with differentactivities may be differentiated from one another based at least on themagnitude of different motion vectors in the sensor data, and withoutcomparing the motion vectors to one or more physical motion profiles.

FIGS. 4, 5, 6, and 7 illustrate the data collected by an accelerometerand a gyroscope on a wearable device as a user is performing differentactivities (gestures), in accordance with some embodiments. For example,parts A and B of FIG. 4 respectively illustrate the magnitudes of theangular velocity vector and the acceleration vector as a user issmoking. As shown in FIG. 4, the magnitudes of the angular velocityvector in part A and the acceleration vector in part B show a temporalcorrelation, as indicated by the circled regions. By comparing themagnitudes of the angular velocity vector in part A to the accelerationvector in part B over different temporal periods, the gesture analysisengine can determine a probability that the user's gesture correspondsto smoking. For example, in FIG. 4, the data may indicate that the userhas taken four cigarette puffs (as indicated by the four circled regionsin part A).

The above analysis may be extended to other types of behaviors oractivities. For example, parts A and B of FIG. 5 respectively illustratethe magnitudes of the acceleration vector and the angular velocityvector as a user is eating. The circled regions in parts A and B of FIG.5 illustrate a correlation between the magnitudes of the accelerationvector and the angular velocity vector as the user is eating.

Parts A and B of FIG. 6 respectively illustrate the magnitudes of theacceleration vector and the angular velocity vector as a user isbrushing teeth. In FIG. 6, although the magnitude of the accelerationvector changes at high frequency (shown in part A), it may be observedthat the magnitude of the angular velocity vector changes at a lowerfrequency. A correlation can still be made because this magnitudepattern of the acceleration vector and angular velocity vector may beunique to teeth-brushing.

Parts A and B of FIG. 7 respectively illustrate the magnitudes of theacceleration vector and the angular velocity vector as a user isdrinking a cold drink. As shown in FIG. 7, there is also a correlationbetween the magnitudes of the acceleration vector and the angularvelocity vector.

Comparing FIGS. 5, 6, and 7, it may be observed that the magnitudes ofthe acceleration vector and the angular velocity vector are different(and vary at different frequencies) between the different activities,and therefore can be used to distinguish among the different activities.In particular, by comparing the magnitudes of the vectors alone acrossdifferent temporal periods, and without comparing the vectors to actualphysical motion profiles (or patterns), the processing power and/or timeneeded to detect a gesture associated with the behavior/activity can bereduced, according to various embodiments described in this disclosure.

In some embodiments, the gesture analysis engine may be configured todetermine a correlation between the magnitudes of the accelerationvector and the angular velocity vector within same/different temporalperiods, so as to determine the probability of the user performing thepredefined gesture. FIG. 8 illustrates the data collected by anaccelerometer and a gyroscope on a wearable device as a user is smoking,in accordance with some embodiments. Parts A and B of FIG. 8respectively illustrate the magnitudes of the acceleration vector andthe angular velocity vector when the user is smoking (for a singlepuff). During a single puff in a smoking episode, the user may firstbring the cigarette to the mouth (hand-to-mouth gesture), inhale (take apuff), remove the cigarette from the mouth (mouth-to-hand gesture), andexhale. In region 802 of FIG. 8, the user's hand may be in a restposition. In region 804, the user may be bringing the cigarette to themouth. In region 806, the user may be taking a puff (inhaling). Inregion 808, the user may be removing the cigarette from the mouth andexhaling. In region 810, the user's hand may be again in a restposition. As shown in FIG. 8, the magnitudes of the acceleration vectorand the angular velocity vector may show a correlation for eachsubmotion of the smoking gesture.

Statistical Analysis of Sensor Data to Determine Probability of UserPerforming a Predefined Gesture

The gesture analysis engine may be configured to analyze the sensor datawithout comparing the sensor data against one or more physical motionprofile patterns. A physical motion profile pattern as used herein mayrefer to any pattern that has substantially a same profile as acorresponding physical gesture of a user. A shape of the physical motionprofile pattern may be substantially similar to a shape of thecorresponding physical gesture of the user. For example, if a userphysically makes an L-shaped gesture, a corresponding physical motionprofile pattern may have substantially an L-shape.

In some embodiments, the gesture analysis engine may be configured tocalculate a multi-dimensional distribution function that is aprobability function of a plurality of features in the sensor data. Thefeatures may be extracted from the sensor data. The plurality offeatures may comprise n number of features denoted by p₁ through p_(n),where n may be any integer greater than 1. The multi-dimensionaldistribution function may be denoted by f(p₁, p₂, . . . p_(n)).

The plurality of features may be associated with various characteristicsof the predefined gesture. For example, in some embodiments, theplurality of features may comprise two or more of the followingfeatures: (1) a time duration of a submotion during the gesture; (2) themagnitude of the acceleration vector; (3) the magnitude of the angularvelocity vector; (4) the roll angle; (5) the pitch angle; and/or (6) theyaw angle. The submotion may be, for example, a hand-to-mouth gestureand/or a mouth-to-hand gesture. Accordingly, the multi-dimensionaldistribution function may be associated with one or more characteristicsof the predefined gesture, depending on the type of features that areselected and analyzed by the gesture analysis engine. Themulti-dimensional distribution function may be configured to return asingle probability value between 0 and 1, with the probability valuerepresenting a probability across a range of possible values for eachfeature. Each feature may be represented by a discrete value.Additionally, each feature may be measurable along a continuum. Theplurality of features may be encoded within the sensor data, andextracted from the sensor data using the gesture analysis engine 108.

In some embodiments, two or more features may be correlated. The gestureanalysis engine may be configured to calculate the multi-dimensionaldistribution function by using Singular Value Decomposition (SVD) tode-correlate the features such that they are approximately orthogonal toeach other. The use of SVD can reduce a processing time required tocompute a probability value for the multi-dimensional distributionfunction, and can reduce the amount of data required by the gestureanalysis engine to determine a high probability (statisticallysignificant) that the user is performing the predefined gesture.

In some embodiments, the gesture analysis engine may be configured tocalculate the multi-dimensional distribution function by multiplying thede-correlated (rotated) 1D probably density distribution of eachfeature, such that the multi-dimensional distribution function f(p₁, p₂,. . . , p_(n))=f(p₁)*f(p₂)* . . . *f(p_(n)). The function f(p₁) may be a1D probability density distribution of a first feature, the functionf(p₂) may be a 1D probability density distribution of a second feature,the function f(p₃) may be a 1D probability density distribution of athird feature, and the function f(p_(n)) may be a 1D probability densitydistribution of a n-th feature. The 1D probability density distributionof each feature may be obtained from a sample size of each feature. Insome embodiments, the sample size may be constant across all of thefeatures. In other embodiments, the sample size may be variable betweendifferent features.

In some embodiments, the gesture analysis engine may be configured todetermine whether one or more of the plurality of features arestatistically insignificant. For example, one or more statisticallyinsignificant features may have a low correlation with the predefinedgesture. In some embodiments, the gesture analysis engine may be furtherconfigured to remove the one or more statistically insignificantfeatures from the multi-dimensional distribution function. By removingthe one or more statistically insignificant features from themulti-dimensional distribution function, a computing time and/or powerrequired to calculate a probability value for the multi-dimensionaldistribution function can be reduced.

Smoking Statistics Example

In some embodiments, the gesture analysis engine may be configured toanalyze the sensor data to determine a probability that a user smoking.The probability may be determined based in part on a magnitude of anacceleration vector and/or an angular velocity vector in the sensordata, and without comparing the acceleration vector and/or the angularvelocity vector to one or more physical motion profiles. In thoseembodiments, the gesture analysis engine may be configured to analyzeone or more features in the sensor data to determine a probability ofthe user taking a cigarette puff. The features may comprise at least oneof the following: (1) a time duration that a potential cigarette isdetected in a mouth of the user; (2) a roll angle of the user's arm; (3)a pitch angle of the smoker's arm; (4) a time duration of a potentialsmoking puff; (5) a time duration between consecutive potential puffs;(6) number of potential puffs that the user takes to finish smoking acigarette; (7) the magnitude of the acceleration vector; (8) a speed ofthe user's arm; (9) an inhale region corresponding to an arm-to-mouthgesture; and (10) an exhale region corresponding to anarm-down-from-mouth gesture.

The gesture analysis engine may extract the features from the sensordata and insert them into a mathematical function to obtain theconfidence (0-100%) level for which these features match a smokinggesture. If the confidence level is high, the gesture analysis enginemay determine that the user has smoked a cigarette. The mathematicalfunction represents the user statistics. Different users have differentstatistics and functions. A mathematical function may be represented byits polynomial coefficients (a's). Accordingly, the function may bedefined by a set of numbers (a's). For example, in the equation showbelow, P is the function, x is the feature inserted into the function,and a's are the coefficients that represent the function.

${{P(x)} = {{\sum\limits_{i = 0}^{n}\; {a_{i}x^{i}}} = {a_{0} + {a_{1}x} + {a_{2}x^{2}} + \cdots + {a_{n - 1}x^{n - 1}} + {a_{n}x^{n}}}}},{n \geq 0}$

In some embodiments, the gesture analysis engine may be configured tocalculate a multi-dimensional distribution function associated with oneor more smoking characteristics. The one or more smoking characteristicsmay be associated with a user taking a cigarette puff. The gestureanalysis engine may be configured to generate a multi-dimensionaldistribution function for each puff. The gesture analysis engine may beconfigured to determine the probability of the user smoking based on:(1) a number of potential puffs; (2) the multi-dimensional distributionfunction for each potential puff; and (3) a time duration in which thenumber of potential puffs occur. The gesture analysis engine may beconfigured to determine whether a sum of the multi-dimensionaldistribution functions for a number of potential puffs is equal to orgreater than a predetermined probability threshold. For example, thegesture analysis engine may determine that the user is smoking when thesum is equal to or greater than the predetermined probability threshold,and that the user is not smoking when the sum is less than thepredetermined probability threshold. In some embodiments, the gestureanalysis engine may determine that the user is smoking when apredetermined number of puffs have been detected within a predeterminedtime period. For in some cases, the predetermined number of puffs may beat least three puffs, and the predetermined time period may be aboutfive to six minutes. The gesture analysis engine may be configured toanalyze the roll and pitch angles associated with the potential puffs,and discard those puffs whose roll and pitch angles fall outside of apredetermined roll/pitch threshold. The gesture analysis engine may alsobe configured to analyze a time duration between the potential puffs,and discarding the puffs where the time duration falls outside of apredetermined time period.

FIG. 9 is graph of the probability that a user is smoking a cigarette asa function of number of smoking puffs, in accordance with someembodiments. A gesture analysis engine (e.g., gesture analysis engine)may be configured to analyze the sensor data to determine a probabilitythat a user smoking. In some cases, as shown in FIG. 9, the gestureanalysis engine may determine a probability of about 83% that a user issmoking, based on a first puff. The first puff may be an actualcigarette puff. When the user takes a second puff, the gesture analysisengine may determine a probability of about 95% that the user isactually smoking. By the time the user takes a third puff, the gestureanalysis engine may determine a probability of about 99% that the useris actually smoking. Accordingly, the gesture analysis engine may becapable of determining whether the user is smoking based on a number ofpotential detected puffs. In some cases, the gesture analysis engine maybe capable of determining a high probability (e.g., 99%) of the userbased on the first puff

Adaptive Gesture Recognition

As previously described, the input data may comprise user input providedby a user. The gesture analysis engine may be configured adjust theprobability of the user smoking based on one or more user inputs. Theuser inputs may comprise: (1) an input signal indicating that the userdid not smoke; (2) an input signal indicating that the user had smoked;and (3) an input signal indicating that the user had smoked but thesmoking gesture was not recognized or detected.

In some embodiments, the gesture analysis engine may implement analgorithm with a broad statistics that fit the average person (everyone)for a specific type of behavior or gesture. The algorithm can beconfigured to adapt the statistics to a specific person over time. Eachperson may subsequently have a unique configuration file with his/herpersonal statistics, as described below.

For example, the gesture analysis engine may be configured to generate auser configuration file (UCF) for the user based on the analyzed sensordata and the one or more user inputs. Initially, the gesture analysisengine may generate a general UCF. The general UCF may be generic andnon-specific to any user. The general UCF may comprise a list of userparameters associated with smoking. Additionally, the general UCF maycomprise a list of user parameters associated with different activitiesbesides smoking. Examples of those activities may comprise at least oneof the following: standing, walking, sitting, driving, drinking, eating,and leaning while either standing or sitting. The leaning may beassociated with the user's elbow. For example, the user may be sittingand leaning an elbow on an object. In some embodiments, the gestureanalysis engine may be configured to generate a left hand UCF and/orright hand UCF for the user in addition to the general UCF. In someembodiments, the left hand UCF and/or right hand UCF may be incorporatedin the general UCF.

The UCF may be configured to adapt and change over time depending on theuser's behavior. Accordingly, after the gesture analysis engine hascollected and analyzed historical behavioral data of the user for sometime, the gesture analysis engine may generate a personal UCF that isunique to the user, based on changes to the general UCF and/or theleft/right hand UCFs.

In some embodiments, the gesture analysis engine may be configured todynamically change the general UCF, left/right UCF, and/or personal UCFwhen the system detects that the user has not performed a predefinedgesture for a predetermined time period. For example, one or more of theabove UCFs may be dynamically changed when the system detects that theuser has not smoked for a predetermined time period. In someembodiments, the system may send a question or prompt to the user (onuser device and/or wearable device) requesting the user to verify thathe/she has not smoked for the predetermined time period.

In some embodiments, the gesture analysis engine may be configured todetermine whether the user is smoking with a right hand or a left handbased on roll angle and pitch angle obtained from the sensor data. Inthose embodiments, the gesture analysis engine may be configured toupdate the left/right hand UCFs with the left/right hand information ofthe user.

Delivery of Personalized Information

In some embodiments, the gesture analysis engine may be configured toinclude additional features (besides those from the sensor data) intothe multi-dimensional distribution function. For example, thoseadditional features may be associated with the user input, the userlocation, historical behavioral data of the user, and/or social networkinteraction of the user. Those additional features may be extracted fromdata that is not sensor-based.

The gesture analysis engine may be configured to analyze the input datausing one or more statistical functions, and to provide the analyzeddata to the gesture analysis engine. The gesture analysis engine mayinclude natural language processing (NLP) clustering and/or machinelearning capabilities. NLP clustering may be based on machine learning,for example statistical machine learning. The statistical machinelearning may be based on statistical inference. The gesture analysisengine may be configured to learn metrics or characteristics associatedwith a predefined gesture, and to determine a user's progress inmanaging certain types of behavior. This machine learning may beaccomplished the gesture analysis engine by analyzing large corpora ofreal-world input stored in one or more databases. The gesture analysisengine may include statistical models capable of making soft,probabilistic decisions that are based on attaching real-valued weightsto behavioral gesture, depending on its context (e.g., where and whenthe gesture was performed, and under what circumstances). Thestatistical models may be robust to unfamiliar input (e.g. new armmotions of a user) and to erroneous input (e.g. false detection of agesture).

The gesture analysis engine may be configured to deliver personalizedinformation (e.g., recommendations) to a user, by transmitting theinformation to the user device and/or the wearable device. Theinformation may be subsequently displayed to the user on the user deviceand/or the wearable device. A user may rely on the information tomonitor certain types of behavior. In some embodiments, the gestureanalysis engine may be configured to proactively provide guidance toassist a user in managing certain types of behavior, based on the inputdata provided to the gesture analysis engine and the gesture analysisengine.

In some embodiments, the gesture analysis engine may be configured toanalyze the user's social network interaction using an application(e.g., a mobile application) provided by the gesture analysis engine.The application may allow a user to pick a social group within theapplication and to compare his/her performance to other users in thesocial group. The social group may be defined by the users. The users inthe social group may be seeking to manage or control a certain type ofbehavior or habit (e.g., smoking) using the application. The user'sperformance may include the user's successes and/or failures in managingthe type of behavior or habit, compared to other users in the group. Insome embodiments, by extrapolating data in the social group and overdifferent timelines, the gesture analysis engine can more accuratelymonitor the user's progress and provide personalized recommendations tothe user. In some embodiments, the gesture analysis engine may beconfigured to determine the probability of the user performing thepredefined gesture at different times of the day and/or at differentgeographical locations. For example, the gesture analysis engine may beconfigured to determine the probability of the user smoking at differenttimes of the day and/or at different geographical locations. In someembodiments, the gesture analysis engine may be configured to promoteand advertise different products or services based on the accumulativesmoking patterns of one or more users.

In some embodiments, the gesture analysis engine can dynamically providepersonalized recommendations to the user in real-time. The personalizedrecommendations may also be provided at a predetermined frequency, e.g.,every hour, 12 hours, 24 hours, 2 days, 4 days, etc. In some instances,the gesture analysis engine can provide a personalized recommendationbased on the user's behavior, or when there are changes in the user'sbehavior (e.g., when the user is smoking a greater number or a fewernumber of cigarettes compared to before).

In some embodiments, in addition to providing a user with theinformation that the user seeks and will most likely consume, thegesture analysis engine can further provide personalized recommendationsto influence the user's needs and behavior.

During a smoking cessation program, the user's needs and challenges mayvary each day. For example, the user may suffer from anxiety,depression, low spirits, lack of energy, urge to smoke, etc.Furthermore, the user may be influenced by other events such as stressand peer pressure. The gesture analysis engine can be configured to takeinto account the dynamic nature of the user's experiences during thesmoking cessation program. For example, the gesture analysis engine canparametrize the user's behavior and body response characteristics atdifferent timeframes. In some embodiments, the gesture analysis enginecan be configured to determine the user's potential needs, and providepersonalized recommendations based on those potential needs.Accordingly, in some embodiments, the gesture analysis engine may becapable of sentiment analysis, so as to more accurately construe andpredict the user's needs and behavior.

In some embodiments, the analyzed data may be provided by the gestureanalysis engine to a healthcare organization, an insurance company,and/or government agency. One or more of the above entities may use thedata to tailor preventive behavioral programs that promote the healthand well-being of the users.

Methods for Detection of Smoking Behavior

FIG. 10 is a flowchart of a method 1000 of detecting a probability of auser smoking a cigarette, in accordance with some embodiments. First,sensor data may be collected by one or more sensors on the wearabledevice, in real-time, intermittently, at fixed or different frequencies(step 1002). The sensor data may be transmitted to the gesture analysisengine, either directly from the wearable device or via a user device.In some embodiments, the gesture analysis engine may be located on aserver remote from the user device and/or the wearable device.Alternatively, the gesture analysis engine may be located on the userdevice and/or the wearable device. Optionally, various aspects orfunctions of the gesture analysis engine may be implemented using theserver, user device, and/or wearable device. The gesture analysis enginemay be configured to determine a probability of a user smoking based onthe sensor data. The smoking behavior may consist of the user taking oneor more cigarette puffs.

Some or all of the sensors on the wearable sensor may be activated atany time. In some embodiments, a subset of the sensors may be activatedto reduce power consumption of the wearable device. When the gestureanalysis engine and/or user device detects that the user may be taking afirst potential cigarette puff (e.g., with probability <1), the gestureanalysis engine and/or user device may be configured to transmit signalsto the wearable sensor to turn on the other sensors. Some or all of thesensor data may be aggregated and sent in blocks from the wearabledevice to the gesture analysis engine in real-time (either directly orvia the user device).

The gesture analysis engine may be configured to extract a set ofpre-defined features from some or all of the sensor data (step 1004).The gesture analysis engine may be configured to use the set ofpre-defined features to detect a probability of the user smoking, byrating a potential cigarette puff and/or number of puffs (steps 1006 and1008). This may be achieved by analyzing the magnitudes of theacceleration vector and/or angular velocity vector of the hand-to-mouthand mouth-to-hand gestures against certain smoking models.

Based on the rated puffs, the gesture analysis engine can detect whetherthe user is smoking or has smoked a cigarette (step 1010). The gestureanalysis engine may transmit and store smoking-related information intoa database for further and/or future analysis (step 1012). Thesmoking-related information may comprise a duration of a cigarette puff,cigarette type, personal information of the user, location of the user,time of smoking, etc. The smoking-related information may be accumulatedover time and used to generate smoking-behavioral trends of the user.The smoking-related information may be displayed on a graphical displayon the user device (step 1014). The gesture analysis engine can use thesmoking-behavioral trends to improve a confidence level of thestatistical analysis, and to predict when/where a user is likely tosmoke. For example, the gesture analysis engine may analyze thesmoking-behavioral trends to detect hidden correlations betweendifferent parameters in the information. The hidden correlations may beused to predict user behavior and/or habits.

FIG. 11 is a flowchart of a method 1100 of detecting a probability of auser smoking a cigarette, in accordance with some other embodiments.First, data may be collected by an accelerometer and a gyroscope on awearable device (steps 1102 and 1104). In some cases, the data may beadjusted to compensate for the effects of gravitational force (step1106). In some cases, if the data is insufficient to detect a gesture, atime count (e.g., a time duration or frequency) of the sensor datacollection may be increased (step 1108). The sensor data may betransmitted to a user device. The user device may be configured tomonitor sensor data (e.g., accelerometer and/or gyroscope data)collected by the wearable device. The user device may be configured tolook for regions in the signal where one or both signals in theaccelerometer or gyroscope data are below predefined thresholds (step1110). These regions may correspond to ‘suspected puff areas’. If theone or more signals in the accelerometer or gyroscope data are below thepredefined thresholds, the user device may instruct the wearable deviceto increase an event count (step 1112) to increase a sampling frequency(step 1112). If the one or more signals in the accelerometer orgyroscope data are above the predefined thresholds, the collection ofsensor data continues at its previous sampling frequency. Next, the userdevice determines if a time window has expired (step 1114). If the timewindow has not expired, the wearable device may continue to collectsensor data. If the time window has expired, the user device maydetermine whether an event count is greater than a threshold count (step1116). If the event count is less than the threshold count, the timewindow and event count may be reset (step 1120) so that a new set ofsensor data may be collected. If the event count is greater than thethreshold count, some or all of the sensor data may be transmitted tothe gesture analysis engine to detect a probability of the user smoking(step 1118). For example, when a sufficient regions have been detectedin a pre-defined time window (e.g., 10 minutes), the user device maytransmit some or all of the sensor data (including the sensor data ithas already processed) to the gesture analysis engine.

The gesture analysis engine may be configured to evaluate each puffcandidate by comparing it to pre-defined statistics and rate each puff.For example, the gesture analysis engine may extract information fromthe puff signal (e.g., length of time a signal is low, etc.) and compareeach value with a pre-defined empirical statistical model. The model maybe general (the same for all smokers), or specific for each smoker. Theprobabilities are then aggregated into a puff rating. In someembodiments, one or more features may be extracted from the candidatepuff signal and processed using machine learning algorithms to produce apuff rating. The machine learning may comprise supervised-learning,semi-supervised learning or unsupervised learning techniques.

After all the candidate puffs in a time window have been rated, thegesture analysis engine can then determine whether a cigarette wassmoked. For example, the gesture analysis engine may count the puffsabove a certain rating (e.g. 50%) and compare the number of puffs to athreshold (e.g. 4 puffs). If the counted number of puffs is greater thanthe threshold, the gesture analysis engine may determine that the useris smoking a cigarette. Conversely, if the counted number of puffs isless than the threshold, the gesture analysis engine may determine thatthe user is not smoking a cigarette, and may be performing some othergesture.

In some embodiments, the gesture analysis engine may be configured toprocess a whole cigarette signal instead of individually analyzingsingle puffs (e.g., a 10-minute signal instead of an 8-second signal). Awhole cigarette signal may be illustrated in, for example FIG. 4. Asingle puff signal may be illustrated in, for example FIG. 8. Thegesture analysis engine can analyze a pre-defined time window ofaccelerometer and/or gyroscope signals (e.g., the time it may take tosmoke a cigarette may be about 10 min) and detect a user possiblysmoking a cigarette based on the signals. The gesture analysis enginemay be configured to determine a total time that the signal variance isbelow a pre-defined threshold. Alternatively, the gesture analysisengine may be configured to determine a relationship between the timethat the signal variance is below the threshold and the time that it isabove the threshold.

If the gesture analysis engine determines that the time is greater thana pre-defined value, the system may then determine that the user may bepossibly smoking. Once a possible cigarette is detected, the entiresignal can be transmitted to the gesture analysis engine. The gestureanalysis engine may analyze all of the signals (instead of processingeach puff separately) and rate the possible cigarette. This can be doneby transforming the signals into frequency domain and extractingfeatures (e.g., energy in specific frequencies, etc.). The gestureanalysis engine can also process the signals, the signal power, and/orthe signal derivative (rate of change) and extract features therefrom.The features can then be used to rate the possible cigarette. Once thecigarette is rated, the gesture analysis engine can determine whetherthe rating is greater than a pre-defined threshold (e.g. 50%). If therating is above the threshold, then a cigarette is determined to havebeen detected. Once a cigarette is detected, the gesture analysis enginemay try to estimate other puffs based on the first puff sample. In someembodiments, the gesture analysis engine may be configured to extractfeatures from puff candidates as well as from a whole cigarette, todetermine whether the user has smoked a cigarette.

In some embodiments, the gesture analysis engine may be configured toalert and inform the user of changes in behavior, patterns, goalsmatching and other consumption related alerts. For example, the gestureanalysis engine may provide an alert when the user behavior divergesfrom the user's typical behavior or historical behavior. For example,the gesture analysis engine may detect that a user normally smokes 2cigarettes in the morning and 2 cigarettes in the evening. When thesystem detects that the user started smoking 2 additional cigarettes atnoon, the system may send an alert to the user so that the user mayrefrain from smoking the additional cigarettes.

FIG. 12 is a flowchart of a method 1200 of detecting a probability of auser smoking a cigarette, in accordance with some further embodiments.

First, sensor data (e.g., accelerometer data (Acc) and gyroscope data(Gyro)) may be transmitted from a wearable device to the user device,server, and/or gesture analysis engine. The sensor data may betransmitted via one or more wireless or wired communication channels.The wireless communication channels comprise BLE (Bluetooth Low Energy),WiFi, 3G, and/or 4G networks.

As shown in FIG. 12, the sensor data may be transmitted to an algorithmmanager (ALGO Manager). The ALGO Manager may be a module located on thewearable device, user device, server, and/or gesture analysis engine.The ALGO Manager may be configured to extract a portion of the sensordata, and transmit the extracted portion to a filter module(Pre-Filter). The Pre-Filter may be located on the wearable device, userdevice, server, and/or gesture analysis engine. The Pre-Filter may applya filter to the sensor data prior to the analysis of the sensor data. Insome embodiments, analyzing the sensor data may further compriseapplying a filter to the sensor data. The filter may be applied toreduce noise in the sensor data. In some embodiments, the filter may bea higher order complex filter such as a finite-impulse-response (FIR)filter or an infinite-impulse-response (IIR) filter. For example, thefilter may be a Kalman filter or a Parks-McClellan filter. In someembodiments, the filter may be applied using one or more processors onthe wearable device. Alternatively, the filter may be applied using oneor more processors on the user device. Optionally, the filter may beapplied using one or more processors on the server. In some embodiments,the filter may be applied using one or more processors in the gestureanalysis engine.

The filtered sensor data may be provided to the gesture analysis engineas buffered data of a predetermined time block (e.g., in 12 secs block).The buffered data may be received at gesture analysis engine at apredetermined time interval (e.g., every 5 secs). The gesture analysisengine may be configured to detect probabilities of puffs from thebuffered blocks. For example, the gesture analysis engine may beconfigured to detect static areas in candidate puff signals. The staticareas may correspond to regions in the signals where one or both signalsare beneath predefined respective thresholds. These regions maycorrespond to ‘suspected puff areas’. The gesture analysis engine may beconfigured to extract features from the candidate puff signals, and toanalyze the features using statistics (e.g., multi-dimensionaldistribution function) to produce a puff rating. The candidate puffswith the respective puff ratings may be inserted into a puff queue. Thegesture analysis engine may be configured to determine the probabilityof the user smoking based on the puffs in the puff queue. Additionally,the method of FIG. 12 may incorporate one or more of steps previouslydescribed in FIGS. 10 and 11.

Sensor Data Management

In some embodiments, the sensor data may be stored in a memory on thewearable device when the wearable device is not in operablecommunication with the user device and/or the server. In thoseembodiments, the sensor data may be transmitted from the wearable deviceto the user device when operable communication between the user deviceand the wearable device is re-established. Alternatively, the sensordata may be transmitted from the wearable device to the server whenoperable communication between the server and the wearable device isre-established.

In some embodiments, a data compression step may be applied to thesensor data prior to data transmission. The compression of the sensordata can reduce a bandwidth required to transmit the sensor data, andcan also reduce a power consumption of the wearable device duringtransmission of the sensor data. In some embodiments, the datacompression step may comprise calculating a difference between samplesof the sensor data. The difference may be time-based (t) orspatial-based (X, Y, and Z). For example, if there is no difference inthe acceleration magnitudes of a current data sample and previous datasamples, the sensor data is not re-transmitted. Conversely, if there isa difference in the acceleration magnitudes of the current data sampleand previous data samples, only the difference may be transmitted (e.g.,from the wearable device to the user device and/or server). The sensordata may be compressed using a predefined number of bits (e.g., 16bits). For example, 32-bit or 64-bit sensor data may be compressed to 16bits.

The sensor data may be collected at a predetermined frequency. In someembodiments, the predetermined frequency may be configured to optimizeand/or reduce a power consumption of the wearable device. In someembodiments, the predetermined frequency may range from about 10 Hz toabout 20 Hz. In some embodiments, one or more sensors may be configuredto collect the sensor data at a first predetermined frequency when thegesture analysis engine determines that the user is not smoking. The oneor more sensors may be configured to collect the sensor data at a secondpredetermined frequency when the gesture analysis engine determines ahigh probability that the user is smoking. The second predeterminedfrequency may be higher than the first predetermined frequency. In someembodiments, the one or more sensors may be configured to collect thesensor data for a predetermined time duration. Optionally, the one ormore sensors may be configured to collect the sensor data continuouslyin real-time when the wearable device is powered on.

A frequency of the sensor data collection may be adjusted based on thedifferent times of the day and/or the different geographical locations.For example, the frequency of the sensor data collection may beincreased at times of the day and/or at geographical locations where theprobability of the user performing the predefined gesture is above apredetermined threshold value. Conversely, the frequency of the sensordata collection may be decreased at times of the day and/or atgeographical locations where the probability of the user performing thepredefined gesture is below a predetermined threshold value. In someembodiments, one or more sensors in the wearable device and/or the userdevice may be selectively activated based on the probability of the userperforming the predefined gesture at different times of the day and/orat different geographical locations.

In some embodiments, the one or more sensors may comprise a first groupof sensors and a second group of sensors. The first and second groups ofsensors may be selectively activated to reduce power consumption of thewearable device, and to reduce an amount of the collected sensor data.The reduction in the sensor data can allow faster analysis/processing ofthe sensor data, and reduce an amount of memory required to store thesensor data.

In some embodiments, the first group of sensors may be activated whenthe wearable device is powered on. The first group of sensors may beused to determine whether there is a high probability that the user issmoking. The second group of sensors may be inactive prior todetermining whether the user is smoking. The second group of sensors maybe selectively activated when the wearable device is powered on,depending on whether there is a high probability that the user issmoking. For example, the second group of sensors may be selectivelyactivated upon determining that there is a high probability that theuser is smoking. The second group of sensors may be activated to collectadditional sensor data, so as to confirm that the user is smoking,monitor the smoking, and collect additional smoking-related data.

In some embodiments, the wearable device may be configured to operate ina plurality of energy and/or performance modes. The modes may comprise alow power mode in which only some of the sensors are turned on. Thewearable device may have low power consumption when the wearable deviceis in the low power mode. An accuracy of detection of the predefinedgesture may be reduced when the wearable device is in the low powermode, since less information (less amount of sensor data) is availablefor analysis in the low power mode. Additionally, the modes may comprisean accuracy mode in which all of the sensors are turned on. The wearabledevice may have high power consumption when the wearable device is inthe accuracy mode. An accuracy of detection of the predefined gesturemay be improved when the wearable device is in the accuracy mode, sincemore information (greater amount of sensor data) is available foranalysis in the accuracy mode. In some embodiments, the sensor data maynot be analyzed when the wearable device and/or the user device is in anidle mode or a charging mode.

In some embodiments, the sensor data may comprise one or moreparameters. The parameters may comprise at least one of the following:(1) a hand which the user smokes with; (2) a pulse pattern of the user;(3) a location of the user; (4) a wearable device identifier and a userdevice identifier (e.g., MSISDN or Android ID or Advertiser ID orIMEI+mac address); and (5) smoking statistics of the user. The one ormore parameters may be unique to the user, the wearable device, and/orthe user device. In some embodiments, an identity of the user may beauthenticated based on the one or more parameters. The identity of theuser may need to be authenticated to prevent misuse of the wearabledevice and/or user device.

User Interface

In some embodiments, the gesture analysis engine can generate one ormore graphical user interfaces (GUIs) comprising statistics of theuser's behavior. The GUIs may be rendered on a display screen on a userdevice. A GUI is a type of interface that allows users to interact withelectronic devices through graphical icons and visual indicators such assecondary notation, as opposed to text-based interfaces, typed commandlabels or text navigation. The actions in a GUI are usually performedthrough direct manipulation of the graphical elements. In addition tocomputers, GUIs can be found in hand-held devices such as MP3 players,portable media players, gaming devices and smaller household, office andindustry equipment. The GUIs may be provided in a software, a softwareapplication, a web browser, etc. The GUIs may be displayed on a userdevice (e.g., user device 102 of FIG. 1). The GUIs may be providedthrough a mobile application. Examples of such GUIs are illustrated inFIGS. 13 through 19 and described as follows.

Window 1300 of FIG. 13 may be generated after the user device isconnected to the gesture analysis engine and data has been obtained fromthe gesture analysis engine. Window 1300 may be an exemplary windowdepicting various smoking monitoring metrics. In some cases, window 1300may correspond to a home landing page that a user will view first whenopening the application or logging into the application. Window 1300 mayindicate the smoking metrics for the user by day. In the example of FIG.13, window 1300 may display that the user had smoked 4 cigarettes forthat day, with 0% improvement compared to the previous day, spent $1.30on cigarettes, and potentially ‘wasted’ 44 minutes of his/her life bysmoking the 4 cigarettes that day. The amount of time ‘wasted’ may beindicative of a health impact from smoking a number of cigarettes.

In some embodiments, a user may view his/her smoking patterns fordifferent times within a day. For example, as shown in window 1400 ofFIG. 14, a user may have smoked 9 cigarettes for that day, over threesmoking episodes (6+1+2 cigarettes). The piechart in window 1400 furtherillustrates that out of the total smoking for that day, 18% of thesmoking occurred in the morning, 39% occurred at noon, 23% occurred inthe evening, and 20% occurred at night.

In some embodiments, a user may view his/her smoking metrics by week.For example, as shown in window 1500, the barchart indicates that theuser smoked 16 cigarettes on Sunday, 14 on Monday, 19 on Tuesday, 17 onWednesday, 12 on Thursday, 15 on Friday, and 14 on Saturday. It may beobserved that the user smoked the least on Thursday and smoked the moston Tuesday. The piechart in window 1500 further illustrates that 38% ofthe smoking occurred on weekdays and 62% occurred on weekends.

In some embodiments, a user may view his/her smoking metrics by month.For example, as shown in window 1600, the barchart indicates that theuser smoked 102 cigarettes in Week 1, 115 in Week 2, 98 in Week 3, and104 in Week 4. It may be observed that the user smoked the least in Week3 and smoked the most in Week 2. The piechart in window 1600 furtherillustrates that 12% of the smoking occurred in the morning, 45%occurred at noon, 26% occurred in the evening, and 17% occurred atnight.

In some embodiments, a user may set goals in the application. Forexample, as shown in window 1700 of FIG. 17, the user may set a goal oflimiting to 14 cigarettes within a day. This may require the user tospend $4.48 on the cigarettes. Additionally, smoking 14 cigarettes couldpotentially waste 154 mins of the user's life.

In some embodiments, a user may view his smoking behavior compared toother users. For example, as shown in window 1800 of FIG. 18, smoking anaverage of 14 cigarettes per day and an average of 98 cigarettes perweek may place the user in the 6^(th) percentile within the group ofusers.

In some embodiments, a user may view various metrics associated with hissmoking patterns. For example, window 1900 of FIG. 19 illustrates thatthe user had smoked a total of 425 cigarettes, spent $136 on cigarettes,smoked an average of 17 cigarettes per day, and potentially ‘wasted’ 77hours of his life by smoking. Additionally, window 1900 shows that 18%of the smoking occurred at home, 62% occurred at work, and 20% occurredat other locations.

In the GUIs of FIGS. 13 to 19, different colors and shading may be usedto differentiate the segments from each other. The numbers and words forvarious metrics may be provided in different colors and shades toimprove readability, and to distinguish the metrics from one another.Any color scheme or any other visual differentiation scheme may becontemplated.

In some embodiments, a user may be able to share his smoking metricsinformation with other users or contacts. The user may also select oneor more of social media links in the windows to share the smokingmetrics information with other users (e.g., his network of contacts inthe selected social media). The social media links may comprise links tosocial media such as Facebook™ and Twitter™.

In some embodiments, the gesture analysis engine may be configured toreceive puffs/cigarettes information from a plurality of wearabledevices and/or user devices. Each wearable device and/or user device mayserve as a data node that provides user consumption data to a databaseconnected to the gesture analysis engine. The database may be updated inreal-time with the user's smoking data. The gesture analysis engine maybe configured to generate consumption statistics and determinesmoking-related social patterns. For example, the gesture analysisengine can generate a visual representation of aggregated consumptiondata (e.g., total number of cigarettes smoked by day/week/month). Theconsumption data may further include the market share of each cigarettesbrand, consumption per user gender by cigarette brand, and consumerpreferences. Consumer preferences may include time of smoking bycigarette brand, location of smoking (home/work/driving/other), smokingfrequency (per event, per time, per person), consumption per capita, andcorrelation of smoking with consumption of other products (such ascoffee). The gesture analysis engine can also analyze the consumptionstatistics to determine consumption patterns (correlation) for differentbrands, geography, and/or time periods. In some cases, by havingmultiple inputs of smoking behavior, the gesture analysis engine may becapable of cross-learning and recognizing the correlation/impact betweendifferent smokers, which can help evaluate the optimized paths forsmoking cessation for the user as well as his/her social circle. Forexample, the gesture analysis engine may detect that user X is a leaderin his social circle and cessation of smoking by user X maysignificantly influence others in his social circle to change theirsmoking behavior. Accordingly, the gesture analysis engine may provideadditional incentives to user X to assist him in smoking cessation, sothat the effect can be proliferated across the social circle.

In one or more of the previously-described embodiments, the gestureanalysis system is capable of differentiating between smoking patternsof the moving hand and other movements that are not smoking related. Thealgorithms described herein may be based in part on statisticalanalysis, machine learning, signal processing, pattern recognition, anddetection theory. An algorithm may assume a certain smoking model andtry to detect the smoking of a cigarette based on the model. Thealgorithm may also estimate a different smoking model for each smokerand use the model to detect a specific smoker is smoking.

In some embodiments, the gesture analysis system can analyzegeographical, time-based and user attributes (e.g., age, gender, jobvocation, etc.) cigarette consumption trends by aggregating data from aplurality of wearable devices worn by a plurality of users who smoke.

In some embodiments, the gesture analysis system can be used toimplement a smoking cessation program based in part on cognitivebehavioral psychology, by using constant monitoring of near real timesmoking and goal accomplishment in the program. Using the monitoringsystem, a user can be notified of each cigarette that he/she smoked, andreceive instant notification regarding his/her smoking patterns andinformation on the progress in reaching his/her smoking reduction goals.Real-time generation of smoking alerts can be highly effective forsmoking cessation.

Additionally, by transmitting smoking behavior data to a server forfurther analysis, various pattern recognition algorithms can be used todetermine the required milestone/incentive to be offered to the user inorder to effectively influence his/her smoking habits, which can help tochange the user's smoking behavior and decrease the health risks causedby smoking.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B. It will be understood thatalthough the terms “first,” “second,” “third” etc. may be used herein todescribe various elements, components, regions and/or sections, theseelements, components, regions and/or sections should not be limited bythese terms. These terms are merely used to distinguish one element,component, region or section from another element, component, region orsection. Thus, a first element, component, region or section discussedbelow could be termed a second element, component, region or sectionwithout departing from the teachings of the present disclosure.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” or “includes” and/or “including,” when used in thisspecification, specify the presence of stated features, regions,integers, steps, operations, elements and/or components, but do notpreclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components and/or groupsthereof

While preferred embodiments of the present disclosure have been shownand described herein, it will be obvious to those skilled in the artthat such embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the disclosure. It should beunderstood that various alternatives to the embodiments of thedisclosure described herein may be employed in practicing thedisclosure. It is intended that the following claims define the scope ofthe disclosure and that methods and structures within the scope of theseclaims and their equivalents be covered thereby.

1. (canceled)
 2. A gesture recognition method, comprising: obtainingsensor data collected using a plurality of sensors located on a wearabledevice, wherein the wearable device is configured to be worn by a user;and analyzing the sensor data to determine a probability of the userperforming a predefined gesture, wherein the probability is determinedbased in part on (i) a magnitude of one or more motion vectors in thesensor data and (ii) time of day and a geographical location of theuser.
 3. The method of claim 2, wherein the sensor data comprises thetime of day, the geographical location, and at least one of thefollowing parameters: (1) an active hand which the user uses to performthe gesture, (2) a pulse pattern of the user, (3) one or moreidentifiers of the wearable device, and (4) behavioral statistics of theuser relating to the gesture.
 4. The method of claim 2, furthercomprising: updating the probability of the user performing thepredefined gesture based in part on changes to the time of day and thegeographical location of the user.
 5. The method of claim 4, wherein theprobabilities are updated differently for different times of the day andat different geographical locations.
 6. The method of claim 4, furthercomprising: determining if the probability of the user performing thepredefined gesture is above or below a predetermined threshold value. 7.The method of claim 6, further comprising: adjusting a frequency atwhich the sensor data is collected, based in part on the time of day andthe geographical location of the user.
 8. The method of claim 7, furthercomprising: increasing the frequency at which the sensor data iscollected for select time(s) of the day and geographical location(s)where the probability of the user performing the predefined gesture isabove the predetermined threshold value.
 9. The method of claim 7,further comprising: reducing the frequency at which the sensor data iscollected for select time(s) of the day and geographical location(s)where the probability of the user performing the predefined gesture isbelow the predetermined threshold value.
 10. The method of claim 2,further comprising: selectively activating one or more sensors of theplurality of sensors, based on the probability of the user performingthe predefined gesture at the time of day and the geographical location.11. The method of claim 2, comprising: transmitting, over one or morewireless or wired communication channels, a portion or all of the sensordata from the wearable device to at least one of (1) a user device and(2) a server that are configured to analyze the sensor data.
 12. Themethod of claim 11, further comprising: storing the sensor data in amemory on the wearable device when the wearable device is not incommunication with the user device or the server.
 13. The method ofclaim 12, further comprising: transmitting the sensor data from thewearable device to at least one of (1) the user device and (2) theserver when communication between the wearable device and at least oneof (1) the user device and (2) the server is re-established.
 14. Themethod of claim 11, further comprising: applying a data compression stepto the sensor data, in order to reduce at least one of (1) a bandwidthrequired to transmit the sensor data and (2) a power consumption of thewearable device during the transmission of the sensor data.
 15. Themethod of claim 14, wherein the data compression step comprisescalculating a time-based difference between samples of the sensor dataalong different axes of measurement of one or more inertial sensors,wherein the plurality of sensors comprises the one or more inertialsensors.
 16. The method of claim 15, wherein the time-based differenceis transmitted from the wearable device to at least one of (1) the userdevice and (2) the server.
 17. The method of claim 14, wherein the datacompression step comprises using a predefined or reduced number of bits.18. The method of claim 2, wherein analyzing the sensor data comprisesapplying a filter to the sensor data, and wherein the filter is selectedfrom the group consisting of a finite-impulse-response (FIR) filter, aninfinite-impulse-response (IIR) filter, Kalman filter, and aParks-McClellan filter.
 19. The method of claim 3, further comprising:authenticating an identity of the user based on the time of day, thegeographical location of the user, or one or more of the parameters. 20.The method of claim 2, wherein the plurality of sensors comprises one ormore of the following: a magnetometer, a heart rate monitor, a globalpositioning system (GPS) receiver, a barometer, an external temperaturesensor, a microphone, a skin temperature sensor, a capacitive sensor, asensor configured to detect a galvanic skin response, an imaging device,a proximity sensor, an altitude sensor, an attitude sensor, a humiditysensor, a vibration sensor, an infrared sensor, or an audio sensor. 21.A system for implementing gesture recognition, comprising: a memory forstoring sensor data collected using a plurality of sensors located on awearable device, wherein the wearable device is configured to be worn bya user; and one or more processors configured to analyze the sensor datato determine a probability of the user performing a predefined gesture,wherein the probability is determined based in part on (i) a magnitudeof a motion vector in the sensor data, and (ii) time of day and ageographical location of the user.